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Model checking verifies that a model of a system satisfies a given property, and otherwise produces a 
counter-example explaining the violation. The verified properties are formally expressed in temporal 
logics. Some temporal logics, such as CTL, are branching: they allow to express facts about the 
whole computation tree of the model, rather than on each single linear computation. This branching 
aspect is even more critical when dealing with multi-modal logics, i.e. logics expressing facts about 
systems with several transition relations. A prominent example is CTLK, a logic that reasons about 
temporal and epistemic properties of multi-agent systems. In general, model checkers produce linear 
counter-examples for failed properties, composed of a single computation path of the model. But 
some branching properties are only poorly and paitially explained by a linear counter-example. 

This paper proposes richer counter-example structures called tree-like annotated counter-examples 
(TLACEs), for properties in Action-Restricted CTL (ARCTL), an extension of CTL quantifying 
paths restricted in terms of actions labeling transitions of the model. These counter-examples have 
a branching structure that supports more complete description of property violations. Elements of 
these counter-examples are annotated with parts of the property to give a better understanding of their 
structure. Visualization and browsing of these richer counter-examples become a critical issue, as the 
number of branches and states can grow exponentially for deeply-nested properties. 

This paper formally defines the structure of TLACEs, characterizes adequate counter-examples 
w.r.t. models and failed properties, and gives a generation algorithm for ARCTL properties. It also 
illustrates the approach with examples in CTLK, using a reduction of CTLK to ARCTL. The proposed 
approach has been implemented, first by extending the NuSMV model checker to generate and export 
branching counter-examples, secondly by providing an interactive graphical interface to visualize and 
browse them. 

1 Introduction 

Model checking is a verification technique that performs an exhaustive search among states of a given 
finite-state machine to verify that this model satisfies a given property, expressed in temporal or richer 
modal logics 0. Some of these logics, such as CTL are branching: they express facts about the 
computation tree of the model. This branching aspect is even more critical when dealing with multi-modal 
logics. The most common example is CTLK, a temporal-epistemic logic reasoning about time and 
knowledge in multi-agent systems ifTSll . Both temporal and epistemic information are captured as different 
relations over states of the model and properties express facts about all these relations. 

A major benefit of model checking is the capability to generate a counter-example when a property 
is not satisfied. Unfortunately, most of the current state-of-the-art model checkers only return linear 
counter-examples while, in general, branching logics need branching counter-examples 13. 
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Let's take the example of Alice and Bob, where Alice randomly picks a number between 10 and 
100 and Bob has to guess whether the number is prime or not. At each step, Bob can ask Alice whether N 
is divisible by another number m. Based on Alice's answers. Bob has to say whether is prime or not. 

This problem can be modeled into a multi-agent system with Alice and Bob as agents; in such a model, 
N would be undisclosed to Bob. Such a model can then be model checked to verify that Bob always finally 
knows whether N is prime or not; this property can be expressed in CTLK as AF {Ksob Pn V Kgo^ --Pn)-! 
where is true in a state in which N is prime. We say that Bob knows p, written K^oh P. in a state s if p 
is true in all states that are undistinguishable from s for Bob. 

This property is not verified by a model that allows Bob to ask only three questions, and a model 
checker checking this property would return a counter-example. An adequate counter-example for this 
property is not a single computation path: it has to show a path composed of states in which Bob does 
not know whether N is prime, i.e. it also has to show, for each state of this path, another reachable 
undistinguishable state where N is prime and another one where A'^ is not. 

Figure [T] gives a tree-like annotated counter-example explaining why the given property is violated. It 
corresponds to a scenario where = 19, Bob asks whether A^ is divisible by the three first prime numbers 
(2, 3 and 5) and Alice answers negatively each time. The state labels are the values of A^, which Bob does 
not know. The memory of Bob, used to remember Alice's answers, is not shown. The wavy transitions 
link together states that are undistinguishable by Bob and arrowed transitions are temporal ones. States 
are annotated with the properties they satisfy and fi^ansitions are annotated with properties they explain. 

The main branch, composed of bold states, explains how Bob asks his three questions and does not 
know whether A'^ is prime or not. For each state of this main path, two states that are undistinguishable by 
Bob from the main state are given to show that Bob does not know that A'^ is prime (right state) nor that it 
is not (left state). Furthermore, dashed states show that each of these states are reachable from an initial 
one. The highlighted path corresponds to the linear counter-example that a standard model checker would 
provide. 




Figure 1 : A tree-like annotated counter-example for AF {Ksob Pn V Ksob 

This paper proposes branching structures, called tree-like annotated counter-examples (TLACEs), 
that are suitable for explaining violations of branching logic properties. Furthermore, each state of 



S. Busard and C. Pecheur 



41 



these counter-examples is annotated with the part of the (negated) property that it satisfies. These 
counter-examples are defined in the framework of Action-Restricted CTL (ARCTL), a branching-time 
logic with action-labelled transitions |.17j . and their utility is illustrated in the framework of CTLK, a 
temporal-epistemic logic that can be reduced to ARCTL lfT4l . 

Tree-Hke annotated counter-examples are built upon tree-like counter-examples as defined by Clarke 
et al. ITjI, which provide full counter-examples for the universal fragment of ft)-regular logics. These 
counter-examples combine cycles and finite paths in a specific, tree-like structure. Our tree-like annotated 
counter-examples extend the notion of tree-like counter-examples to ARCTL and annotate the states with 
formulas they satisfy to give a better understanding of the violation. 

TLACEs take inspiration from the work of Rasse 1*20 1. Rasse presents branching counter-examples for 
CTL interpreted over states of LTSs. Furthermore, the states of the counter-examples are annotated with 
the sub-formulas they explain. This notion of counter-examples is close to tree-like annotated counter- 
examples. Nevertheless, Rasse does not provides a way to generate counter-examples and TLACEs for 
ARCTL are more general since they are applicable to richer modal logics. 

We have extended the state-of-the-ait symbolic model checker NuSMV PI to generate and export 
tree-like annotated counter-examples for ARCTL properties. One of the drawbacks of these richer counter- 
examples is their size and complexity, which can be polynomial in the number of states of the system and 
exponential in the length of the checked formula in the worst case. We have developed a tool that takes 
TLACEs generated by our extended NuSMV and provides a graphical interface to visualize and browse 
them. These tools have been used to provide a first assessment of the approach. They have been designed 
to generate and visualize TLACEs. Thanks to their parameters and functionalities, they allow the user to 
limit and manage the size of the counter-example. 

The contributions of this paper are: 

• the definition of tree-like annotated counter-examples; 

• the design of an algorithm generating these counter-examples; 

• the implementation of this algorithm in NuSMV; 

• the design and implementation of an interactive visualization tool for browsing these counter- 
examples. 

This paper is structured as follows. Section [2]reminds the syntax and semantics of ARCTL and CTLK. 
Section [3] defines tree-like annotated counter-examples and explains how to generate them. Section]?] 
illustrates the approach with the example of the dining cryptographers. Section ]5] describes the extension 
of NuSMV and the visualization tool. Section ]6]presents the evaluation of these tools. Finally, Section]?] 
presents related work. 

2 Temporal and Epistemic Logics 

This section presents the logics used in this work, ARCTL and CTLK. It first presents the syntax and 
semantics of ARCTL [ 17], then presents CTLK, a temporal-epistemic logic, and describes a reduction of 
CTLK to ARCTL. 

2.1 Action-Restricted CTL 

Action-Restricted CTL is an extension of CTL applied to systems with labelled states and actions, where 
temporal operators are augmented with propositional expressions over actions, expressing properties of 
particular paths of the system. 
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In addition to the usual logical connectives (-■, V, A, =^ and <;=^ ), ARCTL provides temporal 
operators, composed of an action-restricted path quantifier or Aa immediately followed by a path 
operator (X, G, F, U and W). Path operators define path formulas while path quantifiers and logical 
connectives define state formulas. Actions expressions a are composed of actions and logical connectives. 
For example, EqX <p means that there exists a successor reachable through the action a that satisfies ; 
AhG Y means that all states reachable through b actions satisfy 

ARCTL properties are interpreted over the states of a Mixed Transition System (MTS). An MTS is a 
tuple ^ = {S,So,A, T, Ys, ^a) over two sets of atomic propositions Ps and Pa, where 5 is a set of states, 
SqQS are initial states, A is a set of actions, rC5xAx5isa transition relation, and ^ : 5 — )• 2^* and 

: A — )• 2^-* are two functions labeling states with subsets of ^5, and labeling actions with subsets of Pa, 
respectively. These two functions represent the propositions that are interpreted over states and actions, 
respectively. We write s s' for {s,a,s') ^ T . 

A path of ^ starting at is a (finite or infinite) sequence of states and actions w = {sii,ai,s\,a2,S2,...) 
such that Sj w{i) denotes Sj. Yl{^ ,s) is the set of maximal paths in ^ starting at s. = 

(5,50, A, T\a,'fs: ^) is the a-restriction of ^ where T\a = {{s,a,s') € T \ ^ ,a |= a} is the transition 
relation T where only actions a satisfying a are considered. 

We write ^ ,s |= when a state s of an MTS satisfies an ARCTL property 0. Logical connectives 
are interpreted in the natural way. For temporal operators, s satisfies 71 (resp. Aa Ti), where tt is a path 
formula, if and only if there exists a path (resp. all paths) in Yl{^\a:s) satisfying 7i. 

A path w of ^ satisfies X if and only if w{\) satisfies 0; w satisfies F (resp. G 0) iff w{i) satisfies 
for some (resp. for all) /. Finally, w satisfies U i/A iff w{i) satisfies i/A for some / and w{i) satisfies 
for all j < i. ^ W Y is equivalent to (0 U V (G 0). 

In the remainder of this paper, all given ARCTL formulas are considered as reduced to their negative 
normal form: negations are distributed over all operators so that they are only applied to atomic proposi- 
tions. Furthermore, equivalences can be applied to reduce formulas to the following base cases: b, ^b (for 
atomic propositions b), ^ Ay/, <p V Y, E^X 0, E^G 0, Ea[0 U and A^ 7i. This allows a more concise 
presentation of concepts without loss of generality. 

2.2 CTLK 

CTLK is a branching-time epistemic logic mixing knowledge relations and temporal ones [18 |. This logic 
is designed to express facts about time and knowledge of agents in a multi-agent system. This section 
presents the syntax and semantics of CTLK. 

CTLK provides the usual logical connectives together with CTL operators (EX, AG, EF, etc.) and 
the knowledge operator Kag where ag is an agent. It also provides some other epistemic operators for the 
group knowledge Eg, the distributed knowledge T)g and the common knowledge Cg, where g is a group of 
agents, but they are not developed here. Nevertheless, they also can be reduced to ARCTL lfT4l . 

CTLK is interpreted over multi-agent systems, where each agent is aware of the possible behaviors of 
the system and of its own local state but not of the local state of other agents. Fonnally, a multi-agent 
system composed of n agents is a Kripke structure = (S'jS'o, T, ~i , ~„, 1^) where T C 5 x 5 is a 
(temporal) transition relation and ~,- C 5 x 5 are epistemic relations. {s,s') G written s ~,- s', iff s and 
s' are reachable states that share the same local state for agent agj. 

An agent ag, knows in a state s iff holds in all reachable states that are undistinguishable from s 
by agj. Formally, ^j^,s |= Kag- (j) if and only if V/ G 5 : / ~, 5 =^ ^^,s' |= <p. Note that ~, must be 
restricted to reachable states (i.e. T*{So)), capturing the fact that agi knows the global system behavior. A 
witness for a reachable state is thus a reverse execution path back to an initial state. 
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2.3 From CTLK to ARCTL 

Some multi-modal branching logics, i.e. logics dealing with more than one transition relation, can be 
reduced to ARCTL. CTLK is such a logic: a multi-agent system and a CTLK formula can be reduced to 
an MTS and an ARCTL formula, respectively [14|. Given a multi-agent system, the corresponding MTS 
has the same set of states. The set of actions contains actions RUN and BACK, used to label temporal 
and reverse temporal transitions, and one action per agent, to label epistemic transitions. The transition 
relation is an aggregation of the temporal relation, the reverse temporal relation and the epistemic ones, 
using corresponding actions. The labeling of states is augmented with the proposition Init to label initial 
states. Init is used to express the reachability of states: a state is reachable from an initial state iff it 
satisfies ^{back}^ init. 

Formally, given a multi-agent system = (5,5o,r, ~i, ...,~„,^) composed of n agents, the 
corresponding MTS is given by ^ = (5,5(),A, T', 1^) where 

• for all states s,s' G 5 : (i) {s,{RUN},s') G T' iff {s,s') G T; (ii) {s,{BACK},s') G T' iff {s',s) G T; 
(iii) {s, {Agti} ,s')eT' iff s s'; (iv) {s, {Agti \ agi G g] , s') G T' iff Va^,- G g : ^ ~; 

• %[s) = y{s) U {Init} if 5 G Sq, f{s) otherwise; i^A is the identity function. 

To reduce a CTLK formula into an ARCTL one, we use the labels RUN, BACK and Agtj to represent 
a temporal transition, a reverse temporal transition and an epistemic transition of agent agi, respectively. 
Formally, let the function R reduce CTLK formulas into their ARCTL form, R is inductively defined as 

• R{b) = if is a propositional formula; 

. /?(EX 0) = ^{MJN}^ Ri^y, R{EG <t>) = E[]iUN}G /?((/)); /?(E[0 U v^]) = E^nuNjm) U /?(v^)]; 

• /?(Ka, 0) = A{^g;.}X (Reachable R{(j>)), where Reachable is a shortcut for E^^ack}^ init. 

3 Tree-Like Annotated Counter-Examples 

This section presents generic structures called tree-like annotated counter-examples, or TLACEs for short, 
for explaining why a state s of & system does not satisfy an ARCTL property . A counter-example 
explaining a violation of in 5 amounts to a witness explaining satisfaction of -k/) in s. From now on, 
this paper will discuss TLACEs as witnesses of properties rather than counter-examples to avoid carrying 
negations all through. 

A TLACE witnessing is a node composed of a state annotated with the direct sub-formulas of 
that it satisfies. Furthermore, for each existential temporal sub-formula it satisfies (i.e. formulas), the 
node contains a branch explaining the formula. A branch is a list of nodes and actions representing a path 
in the model and witnessing the temporal formula. Formally, tree-like annotated counter-examples ai^e 
defined based on the following grammai- of nodes n and paths p: 

n ■.■.= node{s,{{b \ -^)*},{(E« K : {(A« k)*}) 
p ::= {n,{a,n)*) \ {n,{a,n)* ,a,loop{n)) 

where s are states, b are atomic propositions, a are actions, a are boolean expressions over actions and 
71 are ARCTL path formulas. The loop marker is used to represent a looping path, the marked node is 

'The use of subsets of labels is needed to handle distributed knowledge. See 1 14] for details. 
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the first one of the loop. Given anode node{s,aps,{Eai TCi : pi},ahs), each Ece, 71, : pi is called a branch; 
aps, {Ea, TT;} and ahs are annotations. Let State{node{s,aps,ebs,abs)) = s and First {{no, ai,..., rim)) = 
First{{nQ,ai,...,nm, am+ i,loop{n'))) = no. 

A TLACE node node{s,aps, TT,- : pi},abs) is consistent iff all its paths p,- are consistent and satisfy 
5 = State{First{pi)); a TLACE path {no,ai,...,nm) is consistent iff all its nodes are consistent; a TLACE 
path {no,ai, ...,n,„,aff,+i,loop{n')) is consistent iff {no,ai, ...,nm,am+i,n') is consistent and «' = ?iy for 
some < j <m. We will only consider consistent TLACEs in the sequel, and call TLACE, or witness, a 
consistent TLACE node. 

3.1 Adequate Witnesses 

TLACEs ought to be adequate witnesses for a formula in a state j of a model in a precisely 
defined sense. Given a tree-like aimotated witness n = node {s, aps, ebs,abs), the witness has to satisfy 
the following conditions to be adequate: 

• The witness represents a part of the computation tree of Its paths are execution paths in 

• The atomic propositions annotating nodes of the witness are satisfied in the corresponding states of 
^ and the actions of paths satisfy action formulas of <j). 

• The witness is effectively a witness for <j>. It represents (generally partially) a computation tree 
ensuring ^. 

• The annotations of the witness are coherent with <j>. Branch annotations are sub-formulas of i^. 

The first condition above is formally expressed as n matches ^ — the witness is part of the model — 
while the three last ones are expressed as n explains 0) — the witness explains the property in the 
model. An adequate witness for in ^ of ^ is a witness in .v that matches and explains <p in 

The witness n matches ^ if 5 is a state of ^ and each path in ebs corresponds to a path in ^ such 
that the nodes recursively match their respective states. 

Formally, let ^ = (5,5o,A, 7,^,1^). A node n = {s,aps,ebs,abs) matches ^ iff (i) s e S and 
(ii) V(Ecf,. TT; : Pi) G ebs : pi matches . A path p = («o,ai, ...,«,„) matches ^ iff (i) V/,0 < / < 
m : «, matches ^ and (ii) V/,0 <i<m: State{ni) State{ni+i). A looping path p = {no,a\,..., 
nm,cim+i,loop{n')) matches ^ iff {no,ai, ...,nm,am+i,n') matches ^ . 

The witness n explains (/> in ^ if it has the shape of a witness for ^. This highly depends on the 
structure of . For example, a witness for ^\ A (^2 is a node composed of the annotations and branches of 
two nodes explaining 0i and 02 » respectively. 

Formally, n explains {.Ji , (p) is defined recursively over the structure of (j). This definition is given for 
state formulas <j> by the following two tables. 





n explains {.J^ , 0) iff. . . 


true 


n = node{s, {},{],{}) 


b or -^b 


n = node{s, {0}, {}, {}) and ^,s \= 


01 V02 


n explains 0i) or « explains {^ , 02) 


01 A 02 


n = node{s,aps\ Uaps2,ebsi \J ebs2,abs\ Uabs2) 




and node{s,apsi,ebsi,absi) explains {.J^,^i) 


Ace ^ 


n = node{s, {}, {}, {A« ;r}) 




n = node{s, {}, {£« 7t : p}, {}) and p explains (^,0) 
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p explains (^,0) iff. . . 


E«[0i U 02] 
EaG(j) 


p = {node{s, {},{},{}), a, n[), «i explains (^,0) and ^,a = a 

p = {no,ai, ...,nm), V/,0 < i < m : ni explains (^,(^i), nm explains {^,^2) 

and Vj, I <i<m : ^,ai \= a 

p = {n(),ai, ...,nm,a,„+i,loop{n')), Vi : n,- explains 

and Vj, 1 < / < m + 1 : a, |= a 



Finally, n is an adequate witness for ^,s |= (/> iff State{n) = s, n matches ^ and n explains , (/)). 

Note that universal temporal sub-formulas (i.e. formulas) are not explained: elements of abs are 
just aimotations, i.e. ARCTL formulas. While explaining an E« formula needs only one path, the whole 
system is potentially needed to explain an formula. There is no inherent difficulty in explaining them, 
but it would usually result in a huge, unmanageable structure. 

Tree-like annotated witnesses are full witnesses for the existential fragment of ARCTL. In this 
fragment, only existential path quantifiers are allowed and negations are only applicable to atomic 
propositions. TLACEs are full witnesses in the sense that there exists an adequate witness for J^,s \= ^ 
if and only if is satisfied in the state s of ^ . They adequately describe the satisfaction because they 
provide all the reasonably useful information. On the other hand, because tree-like annotated witnesses do 
not explain universal operators, they are not full witnesses for full ARCTL. 

3.2 Generating Counter-Examples 

This section gives an algorithm to generate tree-like annotated witnesses. This algorithm is described by the 
function explain given below, which takes as arguments a mixed transition system ^ = {S, So, A, T, '^a), 
a state s^S, and a property such that ^ , s\=^, and returns a consistent and adequate tree-like annotated 
witness for ^,s \=(^.To perform this computation, it works recursively on the structure of 

The algorithm uses the sub-algorithms EaGexplain, EaU explain and EaXexplain, which return paths 
in ^ satisfying E^G, EaU and EceX operators, respectively. More precisely, EaGexplain{^ , s,<j),a) 
returns a path {so,ai, ...,Sm) where s = sq, V/,0 <i<m: ^ ,Si |= 0, 3^,0 < k < m : st = Sm and Vi, 1 < 
i<m: Jl ,ai |= a. EaUexplain{^,s,(l),Y,oc) returns a path {so,ai,...,Sm) where s = sq, V/,0 <i <m: 
^,Si \= <j>, ^,Sm \= V^and Vj, I <i<m: ^,ai \= a. EaXexplain{^,s,<j>,a) retums apath (50,01,^1) 
where s = so, ^ ,s\ |= ^ and ,a\ |= a. 

This algorithm is correct, i.e. if ^ ,s \= 0, then explain{^ ,s,(j)) retums a consistent and adequate 
tree-like annotated witness for ^,s |= (j). Its correctness can be proved using induction over the structure 
of <j). Due to space limit, the proof is not developed here, but the intuition is given for the EaU case. 
First, EaUexplain retums a witness path for EalYi U 1//2], so ^,Sm \= ^,Si \= Yi for ' < ^ and 
^,ai 1= a for all /. The construction of p in the /or loop and the following instruction builds a path 
composed of nodes witnessing Yi with a last node witnessing Y2, thus altogether p correctly explains 
EaWi U 1/^2] and the result explains Eal^i U 1/^2]- The result clearly belongs to ^ since EaUexplain 
retums a path in Finally, by constmction, the state of the result is s. 
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Function explain(^, s, ^) 
Data: ^ a Mixed Transition System, s a state of <p an ARCTL property, s.t. ^^,s |= (p. 
Result: a tree-like annotated witness n s.t. State{n) = s,n matches ^ and n explains 0). 

switch (/) do 
case true 
|_ return ?ioi/e(5, {},{},{}) 

case Z?, -1Z7 
|_ return Mo<ie(i',{0 },{},{}) 

case v/'i V 1//2 

if 1= v^i then return explain{^ ,s, Yi) 
|_ else return explain{^,s,\if2) 
case 1/^1 A Y2 

node{s,apsi,ebsi,absi) ^ explain{^ ,s, Xj/i) 
node(s,aps2,ebs2,abs2) ^ explain{^,s, Yi) 
return node{s,aps\ Uaps2,ebsi Uebs2,absi Uabs2) 

case A a n 

|_ return nod e{s, {},{}, {A a n}) 

case EaX y 

{so,ai,si) ^ EaXexplain{^ ,s,Y,Cic) 
|_ return node{s,{},{EaX xjf : {node{so,{},{},{}),ai,explain{^,si,Y))},{}) 
caseE'aM U Yi] 

{so,ai, ...,s,„) ^ EaUexplain{^ ,s, \j/i,V2i Of) 

for / G 0..m — 1 do 

|_ p p + {explain{^,Sj,Yi),ai+i) 
[_ return node{s,{},{Ea[Yi U Yi] ■ p + {explain{^,Sm,Y2))},{}) 
case EaG Y 

{so,ai, ...,Sm) ^ EGexplain{^ ,s, l//", a) 

for / G 0..m— 1 do 

M; ^ explain{^,Si, y) 
if s,- = Sm then n,- 
|_ p ^ p + {ni,ai+i) 
|_ return ?iO(3ff'(i', {}, {iiaG i/a : + (/oo/7(n'))}, {}) 



4 Example: the Dining Cryptographers Protocol 

This section uses the dining cryptographers protocol fSl, a well-known example in temporal epistemic 
logic, to illustrate the applicability of tree-like annotated counter-examples to CTLK and multi-agent 
systems. A model of the protocol is given, a CTLK property violated by the system is presented and the 
corresponding counter-example is described. 

Summarizing Q , the protocol of the dining cryptographers can be described as follows: 
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Three cryptographers at restaurant made an arrangement with the waiter for the bill to be paid 
anonymously. One of them might be the payer or it might be NSA. The three cryptographers 
wonder if NSA is paying but nobody wants to say if she pays. To resolve this problem, the 
following protocol is performed. Each cryptographer flips a coin behind his menu such that 
her right neighbor and she can see the result. Each one then claims aloud whether the two 
coins that she saw are equal or different, stating the opposite if she is the payer. An odd 
number of differences indicates a lier, and then a payer; an even number indicates that NSA is 
paying, assuming that the dinner was paid once. No not-paying cryptographer can tell which 
one of the others is the payer. 
We consider a model composed of three agents a, b and c, representing three cryptographers. Each 
agent knows whether she paid, the result of the coin flips to her left and right and the claims of all agents. 
The protocol is executed in three steps. The initial step determines, for each agent, if she is the payer or 
not, making sure that at most one of them is the payer. Then each agent flips her coin. Finally each agent 
makes her claim, depending on the results of the coin flips and whether she is the payer or not. 

We consider the CTLK property (p = -ta. payer =^ AF (K^ b.payerVKa c.payer). It expresses that 
if a is not the payer, then she will eventually either know that b is the payer or that c is. This property is 
obviously violated by the system as the protocol ensures anonymity of the payer. 

The violation of this property is explained by the tree-like annotated counter-example presented in 
Figure|2] It is a witness for -i0 = -la. payer AEG {->Ka b.payer A ->Ka c.payer), with a path ending with 
a loop (the gray states) composed of states satisfying -iK^ b.payer A -iKa c.payer. Each state explains 
-iKfl b.payer by giving an equivalent state for a satisfying -tb. payer and with a backward path to an initial 
state (and similarly for -iK^ c.payer). 




-^b.payer 



-^c.payer 
EG (^Ka b.payer A c.payer) 

iK(, c.payer 




Figure 2: A counter-example for -ta. payer =^ AF (K^ b.payerVKa c.payer) in the model of the dining 
cryptographers. Straight arrows are temporal transitions, waived arrows are epistemic equivalences for a. 

This counter-example clearly illustrates the need for rich counter-examples. A linear counter-example 
would only give the gray part of the presented counter-example, without the annotations, missing a lot of 
information, and the understanding of the violation would be very hard for the user. This counter-example 
is a good representative of CTLK properties mixing temporal and epistemic operators. 



5 Implementation 

The principles exposed in the previous sections have been implemented in two distinct parts. First, the 
well-known open-source symbolic model checker NuSMV Hi has been modified to generate tree-like 
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annotated counter-examples for ARCTL properties. Second, a new tool called TLACE Visualizer has been 
implemented to visualize and manipulate these counter-example^ An XML format has been designed as 
a transfer syntax between the two tools. 

The modified version of NuSMV implements the generation algorithm presented in Section [X2| The 
implementation is based on the EXexplain, EU explain and EGexplain algorithms already implemented 
in NuSMV and modified to take actions into account. It generates tree-like annotated counter-examples 
and can export them into a custom XML format. 

Technically, the algorithm has been implemented slightly differently than presented in this paper but 
the result is equivalent. The implemented algorithm generates a witness for every ARCTL formula, without 
prior reduction to normal form. Negations are handled within the recursive traversal of sub-formulas. 

The implementation supports two kinds of parameters to limit the amount of generated information. 
The first set of parameters allows to selectively generate branches only for some temporal operators 
(e.g. for EaX but not for EalJ nor E^G). The second parameter limits the maximum depth of generated 
branches, in terms of number of nested temporal operators. 

In terms of output, the original version of NuSMV returns only linear (looping) paths as counter- 
examples; on the other hand, the modified version of NuSMV returns richer information that becomes 
difficult to display in a text format. We developed TLACE Visualizer, an interactive graphical interface 
application for displaying and browsing tree-like annotated counter-examples. The counter-examples are 
loaded from XML files produced by the modified NuSMV and pictured as a graph in the main area of 
the interface. The tool also provides different means to arrange the layout of the graph and explore the 
detailed information associated to each node. A snapshot of the interface is given in Figure [3] 
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Figure 3: A snapshot of the interface illustrating the different feature of the tool. 

The tool automatically lays out the counter-example upon loading, according to a custom layout 
algorithm that takes into account the semantic structure of the counter-example. This representation 
presents the general structure of the counter-example, showing branches and loops. Single states or entire 



^These tools are available on http://lvl.info.ucl.ac.be/Tools/NuSMV-ARCTL-TLACE. 
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subtrees can be dragged around for better readability. To support browsing of larger graphs, branches can 
be folded and unfolded to reduce clutter and selectively show relevant information. 

A side panel displays the values of all variables and annotations along a selected path in the graph, 
in a collapsible hierarchical presentation. All variable values can also be accessed as a pop-up menu on 
each node in the main panel, and variables can be selected for display as part of the node's label, giving 
immediate visibility for a few variables of interest. 

Both our extended NuSMV and the visualization tool currently only support ARCTL logic natively. 
That means that epistemic relations are shown in their ARCTL-reduced form. A desirable future extension 
is to display counter-examples according to their original logic notations. This requires some additional 
engineering but poses no major technical challenge. 

6 Evaluation 

This section first assesses the benefits of the provided browsing facilities to manage complex counter- 
examples in the context of multi-agent systems and CTLK. It then discusses how the approach could be 
extended to handle larger counter-examples and universal witnesses using an interactive, incremental 
generation. 

6.1 Richness and Complexity of TLACEs 

The need for branching counter-examples for CTLK properties has already been illustrated in Section]?] 
with a property violated by the protocol of the dining cryptographers. This example showed that a linear 
counter-example was not enough to fully understand why the model (or the property) was wrong. 

To illustrate the increasing richness and complexity of counter-examples, let's consider the following 
property on the dining cryptographers: cryptographer a will eventually know whether cryptographer 
b knows whether a is paying or not. Let K^^ be a shortcut for Kag (j) VK^g meaning that agent 
ag knows whether (j) is true or not. The above property can then be expressed as AF a. payer, 

meaning that a always eventually knows whether b knows if a is the payer or not. This property is violated 
by the model since, if ft or c is the payer, then a cannot say whether b knows whether a paid or not (if b 
paid, b knows, if c paid, b does not). 

A screenshot of a counter-example for this property is given in Figure]4] The counter-example features 
many different branches, due to the nested operators and the disjunction resulting from K^^ (j). This 
complexity increases when the number of nested epistemic operators increases: the counter-example for 
the property AF a. payer contains 75 TLACE nodes and 37 branches; the counter-example 

for the same property with 8 nested K^^ operators would contain 195 TLACE nodes and 97 branches. 

While this increase still remains linear in terms of the length of the property, theory predicts (and 
experiments confirm) that the number of nodes of a tree-like counter-example for .^,s \= <p can be 
Odi'll'^l) in the worst case, where \S\ is the number of states of the model ^ and |0| the length of the 
violated property. Indeed, for an ARCTL formula of depth D (for example E;„,eG Efj-u^G ... Ef^ueG b), 
the top-level branch in the counter-example may have up to 0( l^l ) nodes, each of which carrying a branch 
with a counter-example of depth D—l, giving a total of Od^j^) nodes. Consequently, the time needed 
to generate a tree-like annotated counter-example is at least in Odi'll'^l) since the tool has to generate 
all nodes of the counter-example. The exact performance depends on the complexity of BDD-based 
re-construction of the execution tree of the counter-example, amortized through the use of memoizing, 
which is beyond the scope of this analysis. 
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TLACE Visual izer 1.1 
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Figure 4: A counter-example for the property AF a. payer violated by the protocol of the dining 

cryptographers, as presented by the tool TLACE Visualizer. The counter-example is expressed in terms 
of MTS and ARCTL. The inner values of states show who is the payer. The transitions labels show 
the transitions types: RUN = 1 for temporal transitions, PAST = 1 for reverse temporal transitions and 
ag.me = 1 for the epistemic transitions of agent ag. 



6.2 Towards Interactive Witness Generation 



As the size of the model and the complexity of the property grow, the generation of a counter-example 
may become intractable. A proposed solution, still to be investigated, is to generate the counter-examples 
in a lazy manner: instead of generating all the information in one batch, the tool outputs an initial state 
or an initial prefix of the counter-example and the user can ask the system to extend the parts of the 
counter-example that are most relevant to his understanding of the reported situation. This interactive, 
incremental approach can also handle witnesses of universal operators: the user will be offered to ask for 
the expansion of selected branches, rather than being provided with the expansion of all branches. 

In such an approach, the Tool plays a game with the User where the Tool tries to show that the property 
is violated while the User tries to show that it is satisfied. The Tool will be responsible of showing 
witnesses for existential operators — by giving adequate branches — while the User will attempt to refute 
witnesses for universal operators (and fail, if the universal property indeed holds). This approach will 
require a two-way interaction between the visualizer and the model checker (NuSMV), through which 
the visualizer will drive the incremental witness generation in the model checker. The model checker 
will obviously have to be extended to support those incremental capabilities. This approach is related to 
game-based model checking, as developed e.g. in ll22lin. 
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7 Related Work 

Other authors propose structures similar to tree-like annotated counter-examples to provide useful infor- 
mation about a violation. 

For example, Gurfinkel and Chechik generate proof-like counter-examples for CTL properties violated 
by Kripke structures 1 12 |. These counter-examples are based on a proof of the violation and are composed 
of states to which parts of the proof are linked. The proof steps are mechanically derived from the 
structure of the property and the counter-example, so a similar result could be produced on TLACEs by 
the visualization tool. Note that this would invert the process, since Gurfinkel and Chechick generate the 
counter-example from the proof and not the other way round. 

Shoham and Grumberg propose a game-based framework for CTL counter-example generation ll2T]| . 
Counter-examples are sub-graphs of the game-graph used to perform model checking. Each node of this 
graph is composed of a state of the model and a sub-formula of the property that it violates. This approach, 
similar to TLACEs and the proof-like counter-examples of [12], is applied to the context of incremental 
abstraction-refinement. The structure of these counter-examples is similar to TLACEs. Nevertheless, 
due to the granularity of the explanation, the number of steps to illustrate the violation — and then the 
number of nodes of the counter-example — is larger than for a TLACE. Such a counter-example for a given 
property is then larger than the corresponding TLACE for , while giving the same information. 

Dong et al. define a framework to explore rich witness structures for modal -calculus called evi- 
dences |l9j. An evidence is a graph with nodes composed of a state of the system and a sub-formula of the 
property. As these evidences can be large, they develop a relational graph algebra to manipulate them, 
and provide an implementation. Evidences have a structure similar to counter-examples of Shoham and 
Grumberg [21,1 . This framework could be adapted to explore multi-modal logics counter-examples like 
TLACEs. 

Meolic et al. propose another model for richer counter-examples |[T6]| . These counter-examples 
are automata accepting all finite linear traces of a given LTS violating a given ACTL (Action-based 
Computation Tree Logic) property. The tackled problem is not the same as the one of this paper since 
Meolic et al. only consider linear traces. Furthermore, they do not annotate their counter-examples. 

Some authors propose complementary approaches to analyze counter-examples and to extract their 
useful information. For example, Jin et al. annotate a linear counter-example into fated and free-will 
segments, representing the parts of the path where the environment can force the system to go to the 
error (fated segments) or where the system performs mistaken behavior and could avoid it (free-will 
segments) iflBll . This approach is complementary to any counter-example generation and could be applied 
to tree-like annotated counter-examples. Other authors like Groce and Visser [11] and Copty et al. (81 
generate and check variations of a found (linear) counter-example to identify the critical parts that 
cause the violation. This approach is also complementary and could in principle be applied to tree-like 
counter-examples . 

Some people have applied SAT solving to verify CTL properties. For example, Penczek et al. describe 
an algorithm to transform an ACTL (the universal fragment of CTL) model checking problem into a 
SAT problem |[T9ll . The idea is to create a set of paths of length k in the model connected together to 
form a branching counter-example. A solution to the SAT problem represents a viable counter-example. 
Nevertheless, they do not explicitly describe how to provide the counter-example to the user. 

In the domain of epistemic logics, MCK [ 10 1 and MCMAS [ 15] are two tools that perform CTLK 
model checking. The first one, MCK, provides some debugging features like the export of the full graph of 
the system or the counter-example resulting from bounded model checking of a property. MCMAS gives 
also some debugging features: it presents a branching counter-example for a violated CTLK property. 
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similar to TLACEs, but these counter-examples are not annotated. It also displays states information that 
can be filtered in terms of agents but does not give browsing features like the ones provided by TLACE 
Visualizer. 

8 Conclusion and Perspectives 

This paper presents a structure, an algorithm and an implementation to represent, generate and explore 
tree-like annotated counter-examples (TLACEs) for Action-Restricted CTL. These counter-examples are 
branching, explaining why an ARCTL property is violated by a model, but also recursively explaining 
why sub-formulas of the negation of the property are satisfied by the model. Furthermore, elements of 
these counter-examples are annotated to help their understanding. While these counter-examples explain 
ARCTL formulas violations, they become really useful to explain violations of richer branching logics 
hke CTLK, that can be reduced to ARCTL. 

The algorithm uses sub-algorithms to generate paths in the model satisfying paiticular temporal 
operators and works recursively to explain why sub-formulas are satisfied by states of these paths. The 
implementation combines an extension of the NuSMV model checker for generating TLACEs and a 
graphical tool for displaying and inspecting them interactively. 

These counter-examples give more infoiTnation about the violation than linear counter-examples 
and give a better understanding about the system. The annotations help the user to understand the 
structure of the counter-example. By nature, such branching counter-examples can become very large and 
their generation is computationally more costly than generating linear counter-examples. The provided 
visualization tool is essential for conveniently and productively inspecting such large structures. 

To be able to handle larger and more complex counter-examples, we are investigating an interactive 
incremental approach, where the tool provides an initial state of the model violating the property and the 
user can ask to expand branches of interest. This approach would also make it possible to provide useful 
witnesses for universal operators, by allowing the user to choose and simulate only selected branches. 
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